This application claims the benefits of Japanese Application Nos. 9-050132, 10-175301 and 10-259362 which are hereby incorporated by reference.
1. Field of the Invention
The present invention relates to an image processing apparatus such as an image pickup apparatus which performs an image pickup using a space pixel offset method and an image take-in apparatus taking a static image as a digital signal, more particularly to an image pickup apparatus which performs an interpolating processing prior to an image superposition and a technology for enlarging a dynamic range of a digital still camera and improving an image quality while reflecting an intention of a photographer.
2. Related Background Art
In general, a plural image sensor type image pickup apparatus using a color separation prism has been incorporated in video cameras and electronic cameras. In order to achieve a high resolution, this plural image sensor type image pickup apparatus adopts an image pickup method so called a space pixel offset method.
The known image pickup apparatus using the space pixel offset method, which was disclosed in Report by Television Technology Society, 17[51], pp. 1-6, 1993, will be described with reference to FIG. 39 and FIGS. 40A to 40G.
Referring to FIG. 39, a dichroic prism A52 is arranged on an optical axis of an taking lens A51, and the dichroic prism A52 separates an incidence light from the taking lens A51 into a mixed light of Rand B components (magenta component) and a light of two G components. The color separation for the G component is a separation by a half mirror, and performed irrespective of a wavelength of the incidence light.
A RB image pickup device A53a is arranged in a travelling direction of the mixed light. A color filter array A52a in which red and blue color filters are arranged in a stripe fashion or checkerwise is applied to a image pickup surface of the RB image pickup device A53a. In the RB image pickup device A53a, the R and B components of an optical image are subjected to a photoelectric conversion.
On the other hand, G image pickup devices A53b and A 53c are individually arranged in a travelling direction of the lights of the two G components, and the two G components of an optical image are subjected to a photoelectric conversion in the respective devices.
Each of pixel outputs of the G image pickup device A53b is indicated by a symbol ◯ of FIG. 40A, and each of pixel outputs of the G image pickup device A53c is indicated by a symbol xe2x97xaf of FIG. 40B. It should be noted that the G image pickup devices A53b and A53c are arranged so as to be offset in vertical and horizontal directions by xc2xd pixel from each other.
A signal processing circuit A54 rearranges the pixel outputs of the G image pickup device A53b (indicated by the symbol ◯ in FIG. 40A) and the pixel outputs of the G image pickup device A53c (indicated by the symbol xe2x97xaf in FIG. 40B) alternately on corresponding superposition lattice points, thus forming a synthesized image as shown in FIG. 40D. It should be noted that symbols xc3x97 indicate positions that have no corresponding pixels.
Moreover, the signal processing circuit A54 performs an interpolating processing for vacant lattice points having no corresponding pixels (symbols xc3x97 in FIG. 40D). As the interpolation method, there have been a upper and lower average interpolation in which the average of outputs of the pixels adjacent to each other in the vertical direction is calculated (see FIG. 40E), a prior value interpolation method in which the interpolation is carried out using the output of the pixel disposed in the immediately prior or left-adjacent position (see FIG. 40F), and a left and right average interpolation method in which the interpolation is carried out by calculating the average of the outputs between the left and right pixels (see FIG. 40G).
By carrying out such an image processing, the image pickup apparatus using the space pixel offset method can achieve a resolution twice that of an image obtained by a single image sensor type image pickup apparatus.
FIG. 41 and FIGS. 42A to 42D are figures for explaining the problems of the prior art by using concrete numerical values for the above described image processing.
As shown in (A) of FIG. 41, assuming that a vertically-striped lattice image is photographed, the vertically-striped image is formed on image pickup planes of the G image pickup devices A53b and A53c. Light receiving cell constituting the G image pickup device A53b correspond to any of bright and dark portions of the stripe, so that the light receiving cell have a quantity of received light amounting to xe2x80x9c0xe2x80x9d or xe2x80x9c4xe2x80x9d. Furthermore, light receiving cells constituting the G image pickup device A53c corresponds to a boundary portion between the bright and dark portions of the stripe, so that all of the light receiving cells have a quantity of received light amounting to xe2x80x9c2xe2x80x9d.
The signal processing circuit A54 replaces the pixel output, which has been subjected to the photoelectric conversion by the G image pickup device A53b, with a position on a synthesized lattice point, thus forming an image shown in (B) of FIG. 41. Moreover, the pixel output of the image pickup device A53c is replaced with a position on the synthesized lattice point, thus forming an image shown in (C) of FIG. 41.
The signal processing circuit A54 synthesizes these two images, thus forming a synthesized image shown in (D) of FIG. 41. The superposition of the two images is conducted by rearranging each of the pixels of (C) in FIG. 41 on corresponding one of the vacant lattice points of (B) in FIG. 41. For example, the pixel xe2x80x9caxe2x80x9d is rearranged on the position of the lattice point xe2x80x9caxe2x80x9d.
Next, the signal processing circuit A54 performs an interpolation processing for vacant lattice points of the synthesized image. For the vacant lattice points of the synthesized image, instead of the interpolation processing mentioned above, an adjacent evenness interpolation processing is sometimes performed. The adjacent evenness interpolation processing is the one in which a weighted sum of the matrix is calculated relative to each of the pixels adjacent to the vacant lattice points to be interpolated and the obtained value is used as the interpolation value. The interpolation results are shown in FIGS. 42A to 42D. It should be noted that FIG. 42A is a figure showing the vertical average interpolation, FIG. 42B is a figure showing the prior value interpolation, FIG. 42C is a figure showing the horizontal average interpolation, and FIG. 42D is the adjacent evenness interpolation. For example, when the interpolation matrix as shown in FIG. 42D is used, each of the pixels located in the positions adjacent to the vacant lattice points vertically and horizontally is multiplied by xc2xc, and the sum of the multiplied values is used as the interpolation value.
Hereupon, when the edge portions of FIGS. 42A to 42D are paid attention to, in all of the interpolation methods other than the vertical interpolation method, image degrading artifacts are generated in the edge portion, and a distortion of the image, which is mosaic-like, occurs. This is because pixel outputs of the G image pickup device A53c that are other image information contribute to the interpolation in addition to the pixel output of the G image pickup device A53b, so that the image degrading artifacts are generated.
Since when a horizontal stripe lattice image is photographed, the horizontal stripe lattice image is obtained by rotating the vertical lattice image by 90 degrees, image degrading artifacts are not generated in the horizontal average interpolation. However, the image degrading artifacts are generated in the vertical average interpolation, so that a distortion of an image, which is mosaic-like, occurs in the edge portion.
Specifically, when the interpolation processing is performed for the image including the vertical stripe lattice image and the horizontal stripe lattice image, or the image containing the edge portion, the conventional image pickup apparatus has posed a problem that the image degrading artifacts are always generated in the edge portion.
In the conventional image pickup apparatus, the image degrading artifacts are removed by performing a smoothening processing for the image using a linear filter. However, the image degrading artifacts cannot be removed perfectly, and in an image pickup apparatus for which a finer and higher image quality is demanded, the generation of the image degrading artifacts in the edge portion cause deterioration in the image quality.
Hereupon, although a low cost camera can be obtained as the image pickup device becomes smaller, an increase in the number of pixels entails increases in individual pixel sizes, and a dynamic range of the image pickup device that has been rather lacking is apt to be more lacking. As a countermeasure for this, a method for using a plurality of image pickup results obtained by varying light exposure has been known.
In Japanese Patent Application Laid-open No. 8-223491, as shown in FIG. 43, a quantity of incidence light onto an image pickup device B01 is made to be different from that onto an image pickup device B02 by a light splitting mirror B05, and an output from one of the image pickup devices B01 and B02 which receives less quantity of the incidence light is amplified. Then, the output of the other image pickup device which receives much quantity of the incidence light is joined with the output of the image pickup device which receives less quantity of the incidence light, thus enlarging the dynamic range.
As another method, a method in which an image is consecutively taken in with different storage time using one image pickup device CCDB001 as shown in FIG. 44 has been known. An operation of this method will be described concretely. An image is taken in with a long exposure time, and the image taken in is temporarily stored in a memory. An image is taken in with a short time, and this image is synthesized on the image that has previously been taken in and stored in the memory, thus forming a synthesized image with a enlarge dynamic range. For example, this method is disclosed in the literature xe2x80x9cDynamic Range Enlarging Method for Car Loadingxe2x80x9d, Electronic Information Communication Society, pp. 1439, October, 1995.
A principle of the dynamic range enlargement is shown in FIG. 45. Two images in which light exposure is changed against a certain object are taken in. At this time, although one of the two images with a larger light exposure is saturated in the high luminance portion (or high brightness portion) portion, the other image with a smaller light exposure is not saturated yet, and it is saturated when the luminance becomes considerably high. Considering the difference between the light exposures of the two images, the image with a large light exposure is used in the low luminance portion, and the image with a small light exposure is used in the high luminance portion, thus superposing the two images. As a result, an image with an equivalently enlarged dynamic range can be obtained.
The luminance difference (or brightness difference) of the object differs depending on a scene. There are a cases where the luminance difference can be covered with a dynamic range of one image pickup device, for example, about 400 times or less, and the luminance difference can not be covered with the dynamic range, for example, several thousand times.
With small luminance difference of the object, if photography is performed at a range where the light exposure of the pair of image pickup devices does not vary so much, a good image with a more inconspicuous boundary of the synthesize can be obtained. On the other hand, with a large luminance difference of the object, if the photography is performed at a range where the light exposure of the pair of image pickup devices greatly vary, a good image with a wide dynamic range can be obtained.
However, in the prior art shown in FIG. 43, since a ratio of the light exposure of the pair of image pickup devices is steady, an image at a steady range of the luminance of the object is always taken in irrespective of the difference of the luminance of the object. For this reason, a good image coping with the variation of the difference of the luminance of the object cannot be obtained.
Moreover, in the prior art shown in FIG. 44, an image is consecutively taken in, so that in photographing a moving object, a deviation of the image or an image-ghosting occurs in the high and low luminance portions.
To solve the above-described problems, the object of the present invention is to provide an image pickup apparatus using a space pixel offset method which is capable of preventing a generation of an image degrading artifacts in an edge portion.
Moreover, the object of the present invention is to always obtain a good image irrespective of a luminance difference of an object.
An image pickup apparatus of the present invention which performs a photoelectric conversion for each of optical images which are individually formed on corresponding one of light receiving planes via an image pickup optical system and light splitting means, thereby generating image information, and performs an image pickup by a space pixel offset method by a plurality of image pickup means which are arranged while offsetting along the light receiving planes, wherein further provided is interpolating means for performing a pixel interpolation for the image information generated by each of the image pickup means and joining space phase of each of the image information, and image synthesizing means for adding pixel outputs corresponding to the image information which are subjected to the pixel interpolation by the interpolating means, thereby forming a synthesized image.
In the foregoing image pickup apparatus, the pixel information are subjected to the interpolation processing only by the pixels of that image information. Therefore, since the pixel interpolation is performed without using the pixel outputs of other image pickup devices, a generation of image degrading artifacts in an edge portion can be prevented, so that a mosaic-like image having no distortion can be obtained.
Moreover, all of the pixels constituting the synthesized image is produced as a result of the addition processing, so that random noises such as thermal noises can be more surely reduced at the time of the pixel addition.
In the image pickup apparatus to which the present invention is applied as described above, since the generation itself of the image degrading artifacts in the edge portion can be prevented, a constitution for removing the image degrading artifacts can be simplified, thus obtaining an image pickup apparatus with a high image quality and resolution.
Moreover, an image taking-in apparatus of the present invention comprises a splitting optical system arranged on an incidence optical axis from an object, the splitting optical system separating an optical path into two directions; first and second image pickup means, the first image pickup means being arranged on an optical axis of one optical path separated by the splitting optical system and the second image pickup means being arranged on an optical axis of the other optical path separated by the splitting optical system; a first selecting means for selectively switching between an automatic mode in which the apparatus automatically sets a dynamic range of a picked-up image to a predetermined range and a manual mode in which a photographer manually sets the dynamic range to a desired range; object luminance (or brightness) range detecting means for detecting an object luminance range that is a luminance range of an object; an AD converting means for performing an AD conversion for outputs of the first and second image pickup means; synthesizing means for synthesizing the outputs of the foregoing first and second image pickup means, which have been subjected to the AD conversion, to output the synthesized output; and control means for setting a storage time of image information of said first image pickup means and a storage time of image information of said second image pickup means in response to the object luminance range detected by the object luminance range detecting means when the automatic mode is selected by the first selecting means.
With such constitution of the image taking-in apparatus of the present invention, in the automatic mode for controlling the dynamic range automatically, an image with a suitable dynamic range can be taken in by setting the storage time for each of the first and second image pickup means in response to a luminance difference of the object, so that a good image in response to the luminance difference of the object can be obtained.
Another image taking-in apparatus of the present invention comprises a splitting optical system arranged on an incidence optical axis of an object, the splitting optical system separating an optical path into two directions and keeping a light quantity separation ratio at a predetermined value; first and second image pickup means, the first image pickup means being arranged on an optical axis of one optical path separated by the splitting optical system and the second image pickup means being arranged on an optical axis of the other optical path separated by the splitting optical system; object luminance range detecting means for detecting an object luminance range that is a luminance range of an object; AD converting means for performing an AD conversion for outputs of the first and second image pickup means; synthesizing means for synthesizing the outputs of the first and second image pickup means, which have been subjected to the AD conversion, to output the synthesized output; and control means for setting a storage time of image information of the first image pickup means and a storage time of image information of the second image pickup means in response to the object luminance range detected by the object luminance range detecting means.
With such constitution of the image taking-in apparatus of the present invention, an image with a suitable dynamic range can be taken in by setting the storage time for each of the first and second image pickup means in response to a luminance difference of the object, so that a good image in response to the luminance difference of the object can be obtained.
Another image taking-in apparatus of the present invention comprises a splitting optical system arranged on an incidence optical axis of an object, the splitting optical system separating an optical path into two directions; first and second image pickup means, the first image pickup means being arranged on an optical axis of one optical path separated by the splitting optical system and the second image pickup means being arranged on an optical axis of the other optical path separated by the splitting optical system; adjusting means for manually setting a dynamic range of a picked-up image to a desired range by a photographer; AD converting means for performing an AD conversion for outputs of the first and second image pickup means; synthesizing means for synthesizing the outputs of the first and second image pickup means, which have been subjected to the AD conversion, to output the synthesized output; and control means for setting a storage time of image information of said first image pickup means and a storage time of image information of said second image pickup means in response to the desired range set by said adjusting means.
With such constitution of the image taking-in apparatus of the present invention, an image with a suitable dynamic range can be taken in by setting the storage time for each of the first and second image pickup means in response to a desired range set by the adjusting means, so that a good image having a desired dynamic range can be obtained.
According to a preferable aspect of the present invention, when the first selecting means selects the foregoing automatic mode, the foregoing control means selects a ratio (T1/T2) of the storage time T1 of the first image pickup means to the storage time T2 of the second image pickup means is selected in response to the object luminance range detected by the object luminance range detecting means.
Moreover, according to a preferable aspect of the present invention, further provided is second selecting means which, in the case where the photographer set the dynamic range of the pickup image to a desired range manually, is capable of selectively switching an extent of the dynamic range.
According to a preferable aspect of the present invention, in the case where the photographer set the dynamic range of the pickup image to a desired range manually, the ratio (T1/T2) of the storage time T1 of the first image pickup means to the storage time (T2) of the second image pickup means is selected in accordance with the extent of the dynamic range selected by the second selecting means.
Moreover, according to a preferable aspect of the present invention, when a rate of a light quantity separated into the first image pickup means is indicated by r1 and a rate of a light quantity separated into the second image pickup means is indicated by r2, the ratio (r1/r2) is assumed to be a value other than 1. For example, the ratio (r1/r2) is a fixed value ranging from 2 to 20. Thus, a deterioration in an image quality due to a very large difference between the storage times of the first and second image pickup means can be prevented while securing the dynamic range of the image taking-in apparatus.
According to a preferred aspect of the present invention, the ratio (r1/r2) is a fixed value ranging from 4 to 10.
According to a preferred aspect of the present invention, further provided is third selecting means which selects a relative relation between the timing of the storage time T1 of the first image pickup means and the timing of the storage time T2 of the second image pickup means at the time of a release operation, and the foregoing control means controls the first and second image pickup means in accordance with the relative relation between the timings selected by the third selecting means. Thus, an image of a moving object with least blur can be obtained, or an image using blur or image-ghosting as expression means can be obtained.
In a preferable aspect of the present invention, when a difference between completion times when the storage times of the first and second image pickup devices terminate is equal to a predetermined value or more, image information outputted from one image pickup means with a shorter storage time is first read out, and held in memory means temporarily. Image information outputted from the other image pickup means with a longer storage time is subsequently read out. The synthesize of the image is performed by the synthesizing means based on the image information stored in the memory means and the image information of the other image pickup means. Thus, a problem of signal deterioration at the time when the image information is kept or hold at the first and second image pickup devices.
In a preferable aspect of the present invention, further provided are processing means for transforming grayscale (or compressing a gradation) of each of image information outputted from the first and second image pickup means, and an image synthesize is performed by the synthesizing means based on the image information that has been subjected to the grayscale transformation by the processing means. Thus, a dynamic range with more large extent can be secured.
According to a preferable aspect of the present invention, the foregoing object luminance range detection means is realized by using the foregoing first and second image pickup devices, and the foregoing storage times T1 and T2 are changed in a state where a ratio (T1/T2) of the storage time T1 of the first image pickup device to the storage time T2 of the second image pickup device is kept at a predetermined value or more, thereby detecting an object luminance range. Thus, a structure of the image taking-in apparatus can be simplified and economical.
According to a preferred aspect of the present invention, the foregoing synthesizing means performs the synthesis at a luminance over-lapping portion where a luminance of image information outputted from the foregoing first and second image pickup devices overlaps, by using a weight function which value changes in accordance with the luminance continuously, and in the case where the foregoing first selecting means selects the foregoing automatic mode, the weight function changes its shape and parameter in accordance with the foregoing object luminance range. Thus, a discontinuity at a boundary produced at the time when images outputted from the foregoing first and second image pickup devices are synthesized at the automatic mode can be lessened.
According to a preferred aspect of the present invention, the foregoing synthesizing means performs the synthesis at a luminance over-lapping portion where a luminance of image information outputted from the first and second image pickup devices overlaps, by using a weight function which changes in accordance with the luminance continuously, and in the case where a photographer a dynamic range of a photographed image set to a desired range manually, the weight function changes its shape and parameter in accordance with the desired range set. Thus, a discontinuity at a boundary produced at the time when images outputted from the foregoing first and second image pickup devices are synthesized at the manual mode can be lessened.
In a preferred aspect of the present invention, the foregoing synthesizing means performs a synthesis by adding image information outputted from the first and second image pickup means forcibly. In this case, since a calculation processing of the weight function can be omitted, a processing for the synthesis can be performed at a high speed. Moreover, since noises are averaged by output images of the first and second image pickup means, the noises are decreased, thus improving an image quality. When a ratio of a light exposure of the first image pickup device to that of the second image pickup device is larger or smaller than 1, a discontinuity of a gradient of the output occurs at a portion where the image pickup device with a larger light exposure is saturated. If the ratio of the light exposure is set selectively to a suitable value, the region can be used as a knee characteristic region, so that an influence of the discontinuity on a final image can be almost removed.
A preferred aspect of the present invention, further comprises a strobe(, or flash or speed light), fourth selecting means for selecting a flash made of the strobe, and strobe light reaching region discriminating means for detecting the region where the strobe light reaches in the image. The foregoing synthesizing means performs a synthesis at a luminance overlapping portion where image outputs outputted from the foregoing first and second image pickup devices by using a weight function which value changes in accordance with a luminance continuously, and the weight function changes its shape and parameter in accordance with results discriminated by the strobe reaching region discriminating means. Thus, an image with a wide dynamic range can be obtained at the time of use of the strobe.
A preferred aspect of the present invention, further comprises a strobe(or flash) and fourth selecting means for selecting a flash mode of the strobe. The foregoing control means controls the first and second image pickup devices so that storage times of the first and second image pickup devices become equal to each other automatically at the time of use of the strobe. The foregoing synthesizing means performs a synthesis by forcibly adding image information outputted from the first and second image pickup devices. Accordingly, a S/N ratio is increased, thus obtaining a good image.
According to the preferred aspect of the present invention, the first and second image pickup devices are respectively color image pickup means capable of taking in a full color image independently.
According to the preferred aspect of the present invention, the color image pickup means has an on-chip color filter. Pixels are arranged in the relatively same position, and colors of the on-chip color filter are arranged in relatively shifted positions.
According to the preferred aspect of the present invention, in the case where the first selecting means selects the automatic mode, an algorithm of the synthesis by the synthesizing means is switched depending on whether or not the object luminance range is broader than a predetermined value.
According to the preferred aspect of the present invention, in the case where a photographer sets a dynamic range of a photographed image to a desired range, the synthesizing means switches an algorithm of the synthesis by the synthesizing means depending on a broadness of the dynamic range selected by the second selecting means.
According to the preferred aspect of the present invention, the foregoing first and second image pickup means are respectively color image pickup means capable of taking a full color image independently. In the case where the first selecting means selects the automatic mode, the synthesizing means synthesizes images using an algorithm which prefers a prevention of a false color and an enhancement of a resolution, at the time when the foregoing object luminance range is narrow. The foregoing synthesizing means synthesizes images by using an algorithm which prefers a dynamic range extension, at the time when the foregoing object luminance range is broad.
According to the preferred aspect of the present invention, the first and second image pickup means are respectively color image pickup means capable of taking a full color image independently. In the case where a photographer sets a dynamic range of a photographed image to a desired range manually, the synthesizing means synthesizes images by using an algorithm which prefers a prevention of a false color and an enhancement of a resolution at the time when a narrow dynamic range is selected by the second selecting means. When a wide dynamic range is selected by the second selecting means, the synthesizing means synthesizes images by using an algorithm which prefers a dynamic range extension. Thus, a discontinuity at a boundary at the time when the images outputted from the first and second image pickup means are synthesized can be lessened.
In the preferred aspect of the present invention, a strobe and fourth selecting means for selecting a state of use or not (a flash mode) of the strobe are further provided. When the strobe is used, timing of the flash of the strobe is allowed to match with the timing selected by the foregoing third selecting means. When the strobe is made to flash in accordance with the exposing time of the foregoing first and second image pickup means, effects such as a front-curtain synchro-photography and a rear-curtain synchro-photography are obtained.
In another image taking-in apparatus of the present invention, the first and second image pickup means are arranged on an optical axis of one optical path split by the splitting optical system and an optical axis of the other optical path split by the splitting optical system, and color filters with a Bayer array are provided at positions which are relatively shifted from the corresponding position by one pixel, so that colors of two sets of the color filters will be arranged at an interpolation relation, thereby reducing the false color.
A principle of the present invention will be described briefly.
In the image taking-in apparatus of the present invention, the two image pickup means (first and second image pickup devices) are used, and a ratio of a storage time of the first image pickup means to that of the second image pickup is made to be variable, whereby a good image in accordance with a luminance difference of an object and an image according to an intention of a photographer can be obtained.
An apportionment of a quantity of light for the second image pickup device should be small, for example, less than ⅓, and an apportionment of a quantity of light for the first image pickup device should be large, for example, more than ⅔, as long as a decrease in a quantity of light due to splitting of the light into the two image pickup device can be prevented and a difference between storage times (exposure times) T1 and T2 of the first and second image pickup devices is not extremely large. In this case, a ratio of the apportionment of the quantity of the light for the first and second image pickup devices is 2 or more.
When an ordinary use state is selected, that is, when an automatic mode for controlling a dynamic range automatically is selected, a camera selects a ratio of the optimized storage times (T1/T2) automatically in accordance with an object luminance detected by object luminance range detecting means. An image with a wide dynamic range is photographed for an image or object with a wide luminance range, and an image with a high quality is photographed for a narrow luminance range.
Since a case where an intention of a photographer is not reflected may occur in the foregoing automatic mode, a manual mode for manually adjusting the dynamic range can be set. When such manual mode is selected, a wide dynamic range preference mode or a narrow dynamic range mode (image quality preference mode) can be selected.
In the wide dynamic range preference mode, an image with a wide dynamic range and less black and white blur (the black blur means a phenomena in which the tone of dark portion is lost, and the white blur means a phenomena in which the tone of the information beyond 100% is lost or extreme highlight details lost) can be always obtained by suitably synthesizing image information. On the other hand, in the image quality preference mode, trough the dynamic range is equal to that in the case of the single image sensor type image pickup device, an image with a good resolution and less false color can be obtained by suitably synthesizing image information.
For reference, a concrete example of an algorithm for synthesizing the image information outputted from the first and second image pickup devices will be described below.
Assuming that an output value corresponding to the pixel position (x, y) of the image photographed under the exposure condition Ei is Li(x, y) (i=1, 2), the image Lwid(x, y) with a broadened dynamic range is obtained as follows.
for i=1 to i=2 do
for (x, y)=(0, 0) to (Xxe2x88x921, Yxe2x88x921) do
if i=1 then
Lwid (x, y)=L1(x, y) (E2/E1) xcex3
else
if L2(x, y) less than Lsat
then Lwid (x, y)=L2(x, y)
where E2/E1 is a ratio of the light exposure of the first image pickup device to that of the second image pickup device, xcex3 is a parameter of the xcex3 correction, and Lsat is a saturation value of the output of the second image pickup device.
Next, a synthesis method to which another algorithm is applied will be described. In the foregoing method, there is a possibility that a discontinuity occurs in the boundary portion between the regions photographed under different exposure conditions. Therefore, in the overlapping region of the luminance, a method for synthesizing by using a weight function which value continuously changes in accordance with the luminance is adopted. In this case, the image Lwid (x, y) is obtained as follows.
for i=1 to i=2 do
for (x, y)=(0, 0) to (Xxe2x88x921, Yxe2x88x921) do
if i=1 then
Lwid (x, y)=L1 (x, y) (E2/E1) xcex3
else
Lwid (x, y)=f(L2(x, y)) L2(x, y) (E2/E1) xcex3+{1xe2x88x92f(L2(x, y))} Lwid xcex3
where f is a weight function at the time of the image synthesis.